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The use of increasingly more informative evaluations 
of a sinale innovation are documented. The innovation. The New 
Primary Grades Beading* Systen (NRS) , was implemented in a variety of 
settings, and the evaluations track the implementation from the early 
stages of pilot testing through large-scale adoption of the program. 
NBS Is characterized as an eclectic code-breaking approach to 
beginniTig reading instruction as opposed to a whole-word approach. It 
employs letter-sound correspondences, synthetic phonics, analytic 
phonics and linguistic technigues such as displaying differences and 
similarities in spelling patterns^ NBS is both individualized and 
adaptive* Once all the students are at or above Level III and 
receiving instructions from audio cassettes, the teaciier is* free to 
••travel" around the room to diagnose, prescribe, teach and monitor 
student progress and to assess student performance^ Many studies of 
the effectiveness of NBS have been conducted. A sample of, seven of 
the 20 or more studies are reported here. There is one within-school 
study, five within-district across-school studies, and one 
across-district study. In all cases there is pre/posttest information 
cn arade 1 students. Cumulation of evidence across settings allows 
for greater confidence in conclusions drawn from the evaluation. 
(Ruthor/BL) 
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Thia p«per document! the \xbh of increaiingly more Informative eval- 
uations of a single innovations The innovation^ The New Primary 
Grades Reading System (NKS)» was implemented in a variety of settings, 
and the evaluations track the implementation from the early stages of 
pilot testing through large-scale t.doption of the program^ Accompanying 
contrasts were made within a single school, within a school district, 
and across districts^ This approach to evaluation followed the naturally 
occurring process of implementation of the innovation and provides 
convergent evidence as to its effects. The cumulation of evidence 
regarding the success of an innovation across settings allows for greater 
confidence in the conclusions drawn from the evaluation. 
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ITERATIVE EVA7..UATI0NI NRS. AN EXAMPLE 



Gaea Leinhardt and Mary Engel 

Learning Research and Development Center 
University of Pittiburgh 

Since the mid-1960»« educational evaluation haa become a prom- 
inent aspect of educational reform and innovation. Evaluatio..8 of both 
major educational changes and minor curriculum revisions tend to 
cccur after the design and implementation of the innovation rather 
than during it. There are rational and defensible reasons for this. 
First, when an Innovation of any type is supported, it is supported 
on the assumption that some change or improvement is desirable. 
Whether the change represents an improvement is another question. 
Second, the individuals involved in designing, implementing, and 
revising an innovation are not necessarily the same people who design 
and implement an evaluation; therefore, especially in cxirriculum 
design, evaluation efforts are tacked on after development. 

Two important approaches to eval\iation of curricular innovations 
are often proposed and rarely followed. One approach is the design 
and implementation of an experiment with random assignment of units 
(students, classes, or schools) to treatment and tight control to avoid 
deterioration of the design. The results of such a study should be 
clear cut (Boruch & Rindskopf, 1977), The process of true random- 
ization and selection of the correct \init for assignment has proven 
difficult but by no means impossible in education. Homogeneity of 
treatment within groups and separation of treatment between groups 
has been a more difficult problem. A second approach is the design 
and implementation of an explanatory observational study (Cooley, 
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1978). Such a study should start with a model of how tho innovation 
may work and include measures of the independent varlablest While 
this approach it undoubtedly the one that will be followed in most large 
scale fie3d-based evaluations, the process of delineating a model and 
measuring its components requires large numbers of observational 
units (students, classrooms, schools), and it is therefore unlikely 
that during early stages of curriculum evaluation this approach will be 
followed because of cost and complexity of administration* 

The purpose of this paper is to examine the feasibility of a third, 
somewhat more pragmatic, approach to evaluating curriculum inno- 
vations* To use thiff approach, the innovation is followed through its 
implementation in different educational settings, data are gathered at 
whatever level possible on its effects, and then the total picture of 
program effects is constructed by analyzing the convergent and divergent 
evidence* This approach follows the natural route of program develop- 
ment; that is, programs start at some level, positive "results" at that 
level lead to expansion, and then further assessment is followed by 
further expansion* One can visualize this approach as a set of concen- 
tric rings moving out from a single limited contrast to larger and more 
complex ones* The rings are both the levels of expansion for the 
innovation and the levels of contrast for the evaluation. 

This paper draws together a set of small scale evaluations of an 
educational innovation, the New Primary Grades Reading System (NRS; 
Beck, 1977)* NRS was implemented in a variety of settings starting 
in 1973* The settings were of increasing physical distance from the 
developers and tended to have decreasing intervention from the 
implementation staff* At some point in each new implementation 
situation, some evaluative data were collected. This paper describes 
the NRS innovation and presents a chronological overview of its imple- 
mentation and the type of contrasting information available* The 



ovaluatlont aro doacribod as moving with the Implomontfttions Irom 
hiitorlcal withln-ichool contrasts to moro Uxlng concurrent across* 
district contrasts. Tho s<it oi studios is loUowed by a synthesis of tho 
impact of NRS on classroom processes and student achievement. 

Background 

Before exploring the details of the Individual studies that evolved 
from the development and implementation of NRS, a tlescription cf the 
reading system, how it operates in t^e classroom, and a brief history 
of its development are reviewed. 

The NRS Approach 

NRS is characterized by its developers as an eclectic code-breaking 
approach to beginning reading instruction as opposed to a whole-word 
approach (Beck, 1977; Beck, Note 1). Children are taught by receiving 
explicit instruction about letter-sound correspondences that can be 
used ui attempting to read the word. One way in which children are 
taught to road unfamiliar words is by using synthetic phonics, or putting 
sounds together to form words* The strategy developed for use in 
putting the sounds together is called the blending chain. A second way 
children are taught to approach unfamiliar words in NRS is to explore 
the word for its parts, or, more generally, to use analytic phonics. 
NRS also teaches children about linguistic techniques that can be used 
in learning to read words. Words and text within NRS are displayed 
so that differences and similarities in spelling patterns are evident 
to the children. 

The developers of NRS emphasize its individualized, adaptive 
structure. The reading system is individualized in that students, after 
completing the first two levels of the curriculum, work through the 
curriciQum at different rates. The curriculum is adaptive in that 
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Uftchori are provided with multipU tcAching ■trAte»tei for uao with 
dlfftrtnt student!. The curriculum also provides the students with a 
wide variety of Instructional resources which can be used to adapt to 
their individual needs and interestit. NUS Is also dest,{ned to teach 
children self-management skills, permitting students to work on an 
activity or to move bebveen activities without the supervision of the 
teacher. 

The first two levels of the curriculum are designed to teach self- 
management skills, letter-sound correspondences, and techniques for 
blending those sounds into words. Most of the academic and managerit^l 
instruction in these early levels of iIRS occurs in teacher-led small 
groups. As the first student enters Level m of the curriculum, the 
teacher's role changes: The stude its receive Initial Instruction in 
new content or skills from audio cassettes, and, once all of the students 
are at or above Level m, the teacher is free to ^'travel** around the 
room to diagnose, prescribe, teach, and monitor student progrer.s and 
to assess student performince. 

Prescriptive activities assigned by the teacher must be completed 
by the student. Each child knows his/her prescription for the day by 
looking at a prescription sheet that is sequenced within the child's own 
consumable workbook. Prescriptive activities also include teacher - 
led instruction, cassette-led instruction, end tests or progress checks. 
In addition to prescriptive activities, children are assigned or select 
reinforcement and fluency-building activities. These activities may 
include reading a story written to be appropriate for the level in which 
the chUd is working, playing spcclaUy designed games, reading a story 
in a teachcr-led group situation, reading books other than those in 
NRS. or writing stories. 
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Chronoloiiy of NRS 

ShorUy aftor its incsption In 1964, the Learning Reiearch and 
Development Center (t,RDC) became Involved with two public elemen* 
Ury schoolst one located in a major urban area, and the other in a 
•uburb. LRDC was attempting to individualise elemenUry school 
Instruction in reading, mathematics, and science. The research and 
development of indivldualiaed reading instruction began with two basic 
restrictions in mindi that the reading curriculum should emphasise 
code-breaking, and that the LRDC should not attempt to develop its 
own reading curriculum, but should try to modify existing ones 
(Beck, 1977). 

A published, programmed text was in use in reading classrooms 
at both schools in 1970. The text was designed so that the teacher 
presented new content to students; students. In turn, practiced skills 
in their individual workbooks after content was presented. It became 
apparent that a teacher in a classioom with a large number of students 
could not present content to each student, one at a time* The LRDC 
developed, from the instructional strategies avaUablc in the teacher's 
manuml, audio lessons on cassette Upon to Uke some of the introduction 
of new material off theSeachers' hands. The addition of the Upes to 
the curriculum increased the amount of time the teacher had at his/her 
disposal for tutoring students and monitoring their performance. 

A later modification of the published program made by the researchers 
was the introduction of the Early Reading Program (ERP), which incor- 
porated new instructional utrategies for the introduction of letter-sound 
correspondence and the blending of those sounds into words. As these 
and othf^r changes of the program were being tried out in the develop- 
mental schools, the reading project staff began to examine the results 
of standardized reading achievement tests. The results were encour- 
aging, but no statement could have been made linking the changes to 

5 



,9 



modUlcaUQni in tho cuff iculum* ObtarvAiioafl q| e^Udp^n mft4e Uy 
tho sUff Indloatod that m«ny wor# •till hnvlnn dliriculty In Acquiring 
reading sklUj, and tho iitaff continued to bo dis«ati«Cied with the 
curriculum itsoU Cor a number of other reaioni--such ai the inappro. 
priate use of '»flll in the blanks" and the inappropriate introduction 
and practice of similar graphemes (e.g., bandd). For these reasons, 
a decision to develop a new reading curriculum was made* 

The reading staff gradually began implementing NRS in the two 
developmental schools In 1972«1973. Data from the 1973-1974 school 
year at both schools for first grade classrooms revealed that achieve- 
ment for NRS atudents was good as compared to first grade students 
In the previous >oar who had been Uughl with tho patched-up curriculum 
(Beck, 1977; Buehler U Eicholbf rger, Note 3; Eichelberger & Leo, 
Note 2). These data wore difficult to interpret for several reasons. 
Any first grade student who was taught reading using NRS probably 
received some beginning reading instruction in kindergarten using the 
old aystem. Tho reading staff also provided assistance in the class, 
rooms, for thoso children who needed it, before 1973-1974, During 
and after that year, the asalatance was removed, 

As the LRDC reading staff and curriculum developers changed 
their approach from rewriting and revising a beginning reading cur* 
riculum to developing and implementing a new one, the nood for further 
evaluation of the curriculum became more pressing. The study of tho 
implementation and its effects on student achievement become more 
structured as finer and more numerous changes were being made in 
NRS. Strengths and weaknesses of the program were being brought 
to light. Since all evaluations of NRS had been conducted in schools 
with long-term, ongoing relationships with LRDC (i.e. , they were 
using other individualized curricula at the same time), and because 
of the relatively positive results of NRS, it was decided to implement 
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|974^'* •«Kool y««r r«pr«a«nud a major chhnf^p for NM, 
i-tfd pr*^''^**^ impU'fttntttI m thr«« flrat iradtt cUfsroomt 
lU ^ ^ *^ A (B«cH, 1977; w^tklmbura aaa^tf , I9T4; 

Not* ^ thU r«pr««»ntii « fsoond rln| out from th« 

^^''•^ iradori «t School I ut«d NR8 
^%$f^ rffmaUi^d In th« prog mm for Uio ootiro school ^ 

I^^N ^ tM* •'^''^ ^ ftudonti Wfti compared w»th a ^roup of 43 

^ho ^^'^ ^ ^Irit irfd« In 1973-1974 »t School 1 and who hud 
^^^^ ^ j^^^»l r^^^^U Program* Th« LRDC roadlng «Uff wan rtapon- 
^^M^ pr«#*^^*^*'^««rvlc« t«'*liilng ai woll *• (xtonsivo support 
^^Vty^^^^ durl/*' Hrat yoaf of Lmplemonution at School K In 
iL^^Ul the r****^! "Uff waf •vallabU to th0 toachora a.^d aUff at 

^^oO^ ^ * ^Moftloni* "olvo probUmii colUct data on studont 

'^^Ijj,^^^ pi N^^' th« distribution and quality of curricular 

tK^^'^HaU' and •**J*««'Via, tho f Ministration of ttUndardiaod tosts at 
^ tjK . / the ^^^^^ y^ar (McC»slln» Note 4) to the 1974-1975 first 
^ A^^^ ^^d to ^^'^4-1975 second graders (1973-1974 first graders). 

^^ict •*^^***''ators» pr^cipala, teachers, and students had 
foaci*^"* to fi^^t •ttompts at implementation of NRS in 
I M /i (^^^^^^ ^ 9^^^^^ ^975), Administrators and curriculum 
^^tl^l^^g axp^****^ Uiterest l« continuing to use the NRS reading 
^^^.(^1^ dui^' ^® ^975-1976 school year in four first grade 



gr«m In a setting U^t had had no prevtnut 



U av^*^*^*» local p/e«e and TV reactions are cited. This 

***%5^tu^^*tO d*'^*^^© the dl^'eminaaon of results and to document 
»^ KiKv^*^* m int^^^^ ^tween innovation, evaluation, and use of 
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«lii«irMmii in two Dtiirlct A tchooliii thyii ^d^iniK 9inoth«r rinft to 
bnplmitnUlion tVAluaUon from on« ichool lo two iiehooU. Tht 
flrit concurront eonlrait («• opTM»if»4 lo MttorUnl) wni enrrltd oyt 
Uio (oUowiAi yoari Four NRS firal grad«« wtr« conip«ro4 with «ov«n 
tNital ftrtt iradts within tho «amo school district* The NIUI niudanti 
AfAin porfortnod Inittor than th« studonts in th« basiil proRrAms* I^caI 
rotponso to Iho prt»||rAm was v«ry •nthuslAStlr (Wtminslnirt OAH»tl<> » 
1976). 

tn tho spring of 19V6f District A personnel d«cldsd to u«« NM as 
th« mAin roAding curriculum in thoir primAry grAdss* Ths program 
WAS introducvd in aU first grAdo classrooms beginning in S«ptfimb«r, 
1976* Tho program was also used in four second grado and three 
third trade claserooms the same year* 

Accumulations ot positive evidence about the benefits of NRS 
resulted In the adoption of tho reading program by an LRDC-aff iliated 
District 1> Tollow Through site* During 1976-1977 NRS was Implemented 
in two kindergarten Follow Through classrooms and in six first grsde 
Follow Through classrooms* Evaluators also collected dsta from 'Lx 
non-NRS* non-FoUow Through claesrooms for comparison study* 
addin' etill another ring (multiple withln*district comparisons) to the 
implementation ami evaluation* 

Tho developers of NRS saw a need for further evaluation of the 
program that would make comparisons outside of District A* Another 
district (District C) agreed to permit data to be gathered for use in 
such a contrast* Three first grade classrooms in District C acted as 
the control group for eight first grade classrooms in District A. and 
another ring, an across*dlstrlct contrast, was added to the series of 
evaluations* 

Another level of contrast was added to the evaluation of NRS in 
District B in 1977-1978. In addiUon to collectinir HaU on sU NRS 
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and six non-NRS first grade classrooms, data was gathered on six NRS 
and eight non-NRS second grade classrooms. Most second graders 
who had NRS in 1977-1978 had had NRS in 1976-1977 as first graders. 

Specific Studi>. s 

Many studies of the effectiveness of NRS Lave been conducted over 
the years, A saniple of seven of the twenty or more studies are re- 
ported here. All the studies chosen had reasonably clean data, pre- 
and posttests, and verifiable test administrations. No study was 
eliminated from this discussion due to directionality of the findings. 
This section presents the detaUs of each selected contrast and shows 
how each study added to information about NRS. There is one within- 
school study, five within-district across-school studies, and one 
across-district study. In all cases there is pre/posttest information 
on students; in some cases there is descriptive classroom information 
from interviews or questionnaires. 

YTithin-School Contrast; Study 1 

After the pilot test of NRS described in the preceding section, 
the first of the series of evaluations, the within-school contrast, was 
carried out. Data collection and analysis for this study were under 
the control of the implementors (McCaslin, Note 4); raw data are no 
longer available for reanalysis. 

Population. In 1974-1975, the NRS was implemented in three 
first grade classrooms in School 1 of District A. The population of 
the school was approximately 50% white. In 1973-1974, first graders 
at the school had received reading instruction with Scott-Foresman or 
Harper & Row, both basal reading series. These students constituted 
the contrast group. There were 40 students in the NRS group and 43 
students in the basal group. 
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Method , Both groups received the Pintner- Cunningham Primary 
Test (Pintner, Cunningham, & Durost, 1966) in the spring of their 
kindergarten year. Thio test is one of a series of Pintner General 
i^bilities Tests and yields an T.Q score. In the spring of their iirst 
grade year, both groups were tested on four subtests of the Stanford 
Achievement Test (SAT; Madden, Gardner, Rudman, Karlsen, 8c 
Merwin, 1964): Word Reading, Paragraph Meaning, Vocabulary, and 
W"ord Study Skills. 

Analysis and results . Scores for the Pintner-Cunningham did not 
differ significantly between the two groups. T-tests were conducted 
to determine if the NRS group performed significantly better than the 
basal group on the criterion measure. Scores on the SAT were sig- 
nificantly higher for the NRS group on two of the four subtests. Word 
Reading and Paragraph Meaning. Scores on the Word Study Skills and 
Vocabulary subtests were higher for the NRS group, but not signifi- 
cantly so. 

Within-District Contrast: Study Z 

In response to positive reading achievement test results from the 
within- school across -year contrast and to positive reactions by the 
administrators in District A, a study was designed to test the effec- 
tiveness of NRS in a more chcillenging, demanding set of circumstances 
a concurrent within-district across-school contrast. This would help 
to control but not eliminate the challenge to results arii^ing from 
possible historical differences from year 1 to year 2. This contrast 
represents a second ring in the evaluation of the effectiveness of NRS 
and its implementation. 

Population. In 1975-1976, NRS was implemented in four first 
grade classrooms in two Distrvct A schools. The seven remaining 
first grade classrooms within tiie district that received basal instruct 
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tion (Scott-Foresxnan or Harper and Row) were used as the contrast 
group* These seven classrooms were divided among three schools in 
the district. There were 66 students in the NRS group and 121 students 
in the non-NRS group who had pretest and posttest scores* The per- 
centage of minority students enrolled in the first grade had increased 
slightly from the preceding year* 

Method, analysis, and results * All students were tested in the 
fall with the Stanford Early School Achievement Test (SESAT; Madden 
& Gardner, 1969) and in the spring with the newly -modified Stanford 
Achievement Test (Madden, et al* , 1973)* The NRS group scored 
significantly better on the combination of two subtests of the SESAT 
measuring reading skills: Letters and Sotinds, and Aural Compre- 
hension* Subtests of the SAT used for the analysis were Reading: 
Part A, Reading: Part B, and Word Study Skills, the s\im of which 
yields a Total Read score* 

Table 1 

Study 2: 1975-1976 Within-District Contrast 

Achievement Means and Standard Deviations of NRS and non-NRS Groups: Matched Cases 

NRS (66) Non-NRS (121) 

Pretest: SESAT 

Sum of Letters and Sounds 45,82 41 .24 X 

and Aural Comprehension 7.40 7.55 S.D. 
Posttest: SAT 

Total Read: Sum of 108.42 64.79 X 

Reading Part A. Part B 32.19 30.27 S.D. 
and Word Study Skills 

Correlation and Regression (A^^ 1Q7) 

12 3 

1) SESAT Pretest X 

2) NRS{1) .28* X 

3) SAT Total Read .71* .24* X 
Mult/? - .73* SAT - 2.8*X, + lOJ'Xj 

V<.oi 
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Table 1 presents the means and standard deviations of the SESAT 
pretest and the SAT posttest for both the NRS and the non~NRS groups. 
Pretest scores indicate that the NRS group started out slightly ahead 
of the non-NRS group on the combination of the Letters & Sounds and 
Aural Comprehension subtests of the SESAT (4, 6 points) and finished 
the year considerably higher on the posttest (23* 6 points). 

End-of-year achievement (SAT Total Read) was regressed on 
pretest (SESAT Letters and Sounds + Aural Comprehension) and a 
d\uzuny variable code for NRS (1). Table 1 shows the correlation ma- 
trix of the variables in the regression, the Inultiple R, and the raw B 
weights for the regression solution. Both B weights (pretest and NRS) 
are significant (2.8 and 10.7), F (1,184) = 163.09 and 8.77 respectively. 

One aid in interpreting the difference between the twc groups in 
end-of-year achievement comes from the information about instruc- 
tional processes used in reading. In Janviary and February, 1976, an 
interview designed to measure instructional practices in reading was 
administered to the first grade teachers involved in the study. Data 
from the interview can be described using three general descriptors 
or constructs: the opportunity the students have to learn materials 
sampled on the end-of-year performance measures; the structure of 
the instruction for both the teacher and curriculum; and the content of 
reading, or how often students engaged in various types of reading 
activities. Only a selected n\imber of variables from the constructs 
will be discussed here. A summary of these variables is presented 
in Table 2. 

The first set of variables reflects the opportunity students had to 
learn the material on which they were tested. Important measures 
in this construct are: the number of. students enrolled; the number of 
adults in the room during reading instruction; student attendance; the 
nxmiber of minutes allotted for reading per week; and the overlap 
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Table 2 

Meani and Standard Deviations of NRS and ^ Selected Measures 
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between the curricrilum and the criterion test. The average number 
of students enrolled in the NRS classrooms in District A during 1975- 
1976 was lower than the number enrolled in non-NRS classrooms 
(17.8 vs. 21. 1). There were also more adults present for reading 
instruction in a few of the NRS classrooms. Attendance for the two 
groups was very similar (89. 3% for NRS and 91.1% for non-NRS). 
NRS teachers assigned approximately 100 minutes per week more 
reading instruction to their students than non-NRS teachers, and they 
reported a higher overlap between what was presented in the curriculum 
and what was tested on the SAT (75. 6% overlap reported by NRS 
teachers and 48.9% by non-NRS teachers). This means that, in 
general, the NRS students had a greater chance to learn reading than 
the non-NRS students because they had fewer students, more adults, 
more time, and a larger amount of criterion-relevant material. 

The structure construct is represented by five variables: self- 
pacing of students, frequency of use of games, speed with which 
assignments are corrected and returned to students, the percentage 
of unique assignments, and the number of days since the last test. 
Without exception, in the NHS classrooms students worked at their 
own rate, while in non-NRS classrooms most students did not work 
at their own rate. Reading games were used more frequently in NRS 
classrooms. The rate of correction tended to be the same for NRS 
and Ac^A~^RS students. There were large differences between groups 
in the percentage of unique assignments and the number of days since 
the last test: percent unique was 92. 0% for NRS and 16. 6% for non- 
NP j» nximber of days since last test was 5. 0 for NRS and 86. 4 for 
non-NKS. Thus, NRS provides for greater individualization, more 
structure and more feedback. 

The content construct is made up of four variables: frequency of 
student engagement in letter- sound correspondence, frequency of 



letter identification, frequency of blending, and an estimate of the 
match between the curriculum and what the teacher considered to be 
important first grade reading objectives* NRS students were reported 
to have spent slightly more time in letter -sound correspondence, 
letter identification, and blending than non-NRS students. NRS teachers 
reported a closer match between NRS and important reading objectives, 
4. 5 on a 5-point scale; non-NRS teachers rated the match between 
their curriculum and objectives at 3. 6. Thus, the content of the NRS 
curriculum seems to focus on acquiring basic reading skills more so 
than the basal sequence* 

"Within-District Contrast; Study 3 

Population* A second within-district contrast was conducted in 
1976-1977 when an urban school district (District B) from a different 
state than District A adopted NRS for use in six of its first grade 
Follow Through classrooms. The control group consisted of six first 
grade classrooms that received reading instruction with Houghton- 
MiJEflin, a basal series* There were 119 students in the NRS group 
of six classes and 144 students in the non-NRS group for ^hom pre- 
test and posttest data were available* This study replicated Study 2 
in a more economically depressed urban area* In addition, the pre/ 
posttests were selected to more closely reflect the objectives of 
traditional basal series* 

Method* Students in both groups were tested in October, 1976 
with the Murphy -Durrell Reading Readiness Analysis (Murphy & 
Durrell, 1965)* The test contains three subtests* Only the Letter 
Names subtest was used for analysis because performance on the 
Phonemes subtest was strongly influenced by the presence of NRS 
in the classroom prior to pretesting in October* The Learning Rate 
subtest was not used because of inconsistency in administration* 
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End-of-year achievement was measured in May, 1977 by the SAT 
(Madden, et al* , 1973). Subtests used in the analysis were: Reading: 
Part A, Reading: Part B, and Word Study Skills. The three subtest 
scores yield a score for Total Read* 

Analysis and results* Table 3 shows the means and standard devia- 
tions of the Murphy -Durr ell pretest and SAT posttest for both the NRS and 
basal groups. The NRS group started out slightly ahead of the control 
group, possibly because testing of entering abilities did not occur until 
well over a month into the school year. 

A regression analysis indicates that end-of-year achievement test 
scores for the NRS group were significantly higher than for the children 
who used the basal series* End-of-year achievement (SAT Total Read) for 
all students was regressed on initial ability and a dummy variable code for 
NRS (!)• Table 3 shows the correlation matrix of the variables in the regres- 
sion, with the raw B weights for the regression solution. Both B weights, 
pretest and NRS, were significant (2. 1 and 10.6), F (1, 260) = 136. 0» and 
12. 84. 

Table 3 

Study 3: 1 976-1 977 Within-Oistrict Contrast 

Achievsmsnt Means and Standard Deviations of NRS and non-NRS Groups: Matched Cases 

NRS{119) Non-NRS(144) 

Pretest: Murphy-Ourrell 

Letter Names Total 46.73 42.81 ' "x 

5.75 9,15 S.D. 

Posttest: SAT 

Total Read: Sum of 

Reading Part A, Part 8 102.79 83.81 X 

and Word Study Skills 27.78 29.09 S.O. 

Correlation and Regression (iV » 263) 

1 2 3 

1) Murphy-Ourrell Pretest X 

2) NRS{1) .24* X 

3) SAT Total Read .62* .32* X 
Mult/?-. 64* SAT«2.rX, M0.6*X2 

V<.01 

r6 
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Process data for this contrast were collected in the first two 
months of 1977, again through the use of a teacher interview* These 
data are sTiznzxiarized in the second column of Table 2, The opportunity 
construct suggests that the average number of students enrolled in 
NRS classrooms in District B during 1976-1977 was lower (X = 22. 3) 
than the niimber enrolled in the non*NRS classrooms (X = 29. 7). NRS 
classrooms had twice as many adults present during reading instruction* 
The percentage of students present in both settings was equivalent. 
Teachers in NRS classrooms reported that they allocated 500 minutes 
per week fo> reading instruction; non-NRS teachers allocated 85 minutes 
per week more. NRS teache rs reported a 647© overlap between what 
the SAT measured and what was presented in the NRS curricultim; 
non*NRS teachers reported a 42% overlap. Thus, in the second witbln- 
district contrast th^re is again a greater opportunity for students to 
learn to read in NRS classrooms because there are more teachers, 
fewer students, and greater overlap* However, less time is allocated 
to reading indicating that perhaps NRS is more efficient in teaching 
reading than the basal series. 

Fundamental differences between NRS and non-NRS groups appear 
in the structure of the curriculum. Again the NRS group reported a 
higher level oi student self-pacing, and that they use games about 
twice as frequently as non-NRS teachers. NRS teachers also reported 
that they correct student work and return it to the students much 
sooner than their basal counterparts, though neither group reported 
correcting or returning work on-line. The percentage of unique 
assignments on a given day is a measure of individualization in the 
classroom, and NRS classrooms have a much higher percentage of 
unique assignments: 86% for the NRS group and 14.2% for the non- 
NRS group. NRS students tend to be tested more^frequently than 
non-NRS students: The average number of days between tests for 
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the NRS group is 4. 1 days» and 22*7 daya for the non-NR3 group* 
NRS students also tend to score well on those tests* The percentage 
of correct items on a selected test for the NRS group was 95%; for the 
non-NRS group, it was about 75% correct. The results of the structure 
construct are consistent with the other within-district contrast: NRS 
individualizes Instruction and gives more rapid and frequent feedback 
than non-NRS curricula* 

Again NRS teachers report a somewhat greater focus in beginning 
reading content than non-NRS teachers* In general, NRS teachers 
report that they teach letter-sound correspondence more frequently 
than do non-NRS teachers (4*7 for NRS; 4* 2 for non-NRS)* The same 
pattern holds true for identifying letters and blending sounds* The 
rating of the match between the curriculum and what teachers consid- 
ered to be Important first grade reading objectives was high for both 
groups (4* 9 for NRS, 4* 5 for non-NRS)* 

Within-District Contrast; Study 4 

Population * Data collection in District B during 1977-1978 formed 
the basis for the third within -district contrast* Six first grade class- 
rooms received reading instruction with NRS, and six classrooms were 
taught with Houghton-Mifflin, a basal series* Pretest and posttest 
data were collected for 123 NRS students and 133 non-NRS students* 
This study replicates the previous one with the exception of a minor 
change of testing instruments* 

Method. The Comprehensive Tests of Basic Skills (CTBS Level 
B, Form S; CTB/McGraw-HUl, 1973) was administered in October, 
1977 in all 12 first grade classrooms* Five subtests were administered 
Letter Sounds, Word Recognition I, Reading Comprehension, Word 
Recognition II, and language I* A total read score was obtained by 
adding the first four subtest scores* The Language I subtest was not 
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included in the analyses because It is a test of listening and pictorial 
interpretation, not reading. Students were posttested in May, 1978 
with the same form and subtests of the CTBS. Process data were 
also collected during the first months of 1978, this time through the ^ 
use of a questionnaire mailed to the teachers. 

Analysis and results . NRS and non-NRS group means and standard 
deviations for the CTBS are presented in Table 4. Again, the NRS 
group scored higher on the pretest than the non-NRS group. There is 
a 3-point difference between groups on the pretest and a 7-point differ- 
ence on the posttest. £nd-of -year achievement (CTBS Total Read) 
for all students was regressed on initial ability and a dummy variable 
code for NRS (1). Table 4 contains correlations of the variables and 
raw B weights (pretest and NRS), ,both of which are significant (. 77 
and 4.7), F (1, 253) = 93.0 and 7.11. 



Tabu 4 

Study 4: 1977*1978 Within-District Contrast 



Achitvement Means and Standard Deviations of NRS and non-NRS Groups: Matched Cases 

NRS (123) Non>NRSn33) 

Pretest: CTBS Level B 

Total Read 30.49 27.35 X 

12.17 9.62 S.D. 

Posttest: CTBS Level B 

Toul Read 64.11 57.05 X* 

14.15 17.78 S.D. 

Correlation and Regression {.V • 256) 

1 2 3 

1) CTBS Level B Pretest X 

2) NRSd) .14 X 

3) CTBS Level B Posttest .53* .21* X 
Mult/? - .55* CTBS - .77*X, +4.7*X2 



V<.oi 
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Classroom process data (in Table 2) for opportunity show that the 
NRS group again had fewer students enroUed per classroom than the 
non-NRS group and twice as many adults in the room during reading 
instruction. The number of students present in the classrooms was 
equivalent. First grade NRS teachers during 1977.1978 uniformly 
aUotted 450 minutes per week for reading instruction, while non-NRS 
teachers allotted about 362 minutes per week. There exists a rather 
large standard deviation for the non-NRS group on this variable, 
indicating that, as a group, non-NRS teachers allocated reading time 
very differenUy. Unfortunately, the overlap estimate was not collected 
during 1977-1978 for this study. 

The structure construct reveals that the NRS group had more 
classrooms in which students were self -paced. NRS teachers contin- 
ued to report more frequent use of games and that they corrected and 
returned student work more quickly than teachers who used the basal 
series. The percentage of unique assignments was much higher for 
NRS classrooms (81% unique for NRS classrooms; 16. 5% unique for 
non-NRS classrooms); students in NRS classrooms tended to be tested 
more frequently (calculations of the number of days between tests was 
possible for only two of the non-NRS classrooms); and they performed 
better on a selected test than did the non-NRS students. 

Regarding the content construct, NRS students spent more time 
engaged in letter-sound correspondence and letter identification 
exercises, but non-NRS students spent more time in sound blending 
exercises. Both groups of teachers rated the curricula as having a 
close match with what they considered to be important first grade 
reading objectives. 

MTithin-District Contrast; Study 5 

Population. The fourth within-district contrast in this set of 
evaluations was a contra-at at the second giade level in District B. 
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Tabit S 

Study S: 19n-1978 Wilhin^iitrict ConVMt 



NRSni3) Nof>^RS(109) 

PrttKC: CTBSUvtiC 

Total Raad 3442 32.92 X 

16JS6 16.71 S.D. 

Poittift: CTBSLtvalC 

Total Raad 50.82 49.09 X 

1640 17J9 S.D. 

Cof ralation and Ragrattion « 222) 

1 2 3 

1) CT8S Laval C Pratatt X 

2) NRSd) .045 X 

3) CT8S Laval C Putiast .77* .051 X 

UuHR .77* CTBS Poittaft - .79*X^ ♦ MX^ 

V<^1 



Withln»DiBtrictAcro8a "Grade Contraat: Study 5a 

Poptilation * Another contrast arose from the second grade study. 
This analysis was done to determine if there were cumtilative effects 
that were undetected by the simple second grade contrast. This con- 
trast involves tracking NRS students over two years. By springs 1978^ 
there were 112 students who had been in the District B studies since 
the fall, 1976: 83 students in NKS and 29 students in the basal program. 
Clearly the contrast group must be viewed with caution* however, 
because the means for the full first grade group and the two-year 
sample are quite similar as shown in Table 6. 

Method . Achievement test scores for this contrast were compiled 
from the first grade fall 1976 administration of the Letter Names sub- 
test of the Murphy-Durrell and the spring 1978 administration of the 
CTBS Level C to second graders. 
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Table 6 

Study 5«: 1076-1977 «nd 1977-1978 Within-Dittrlct and Acroit-Grada Contrast 
Achievement Meani and Standard Deviations of NRS and non-NRS Groups: Matched Cases 

NRSI83) Non-NRS (29) 

Pretest: Murphy-Durrell 

Letter Names Total 47.78 46.00 X 

4.07 3.79 S.D. 

Posttest: CTBS Level C 

Total Read 52.33 49.83 X 

15.72 17.23 S.D. 

Achievement Means and Standard Deviations of NRS and non-NRS Groups: Unmatched Cases 

NRS Non-NRS 

Pretest: MurphyDurrell 

Letter Names Total 45.97 - 1401 42.02 (^-172} X 

6.97 9.44 S.D. 

Posttest: CTBS Level C 

Tout Read 50.82(^- 113) 49.09(n-109) X 

16.40 17.89 S.D. 

Correlation and Regression (^-112) 

1 2 3 

1) MurphyDurrell Pretest X 

2) NRS(I) .19 X 

3) CTBS Level C Posttest .29* .07 X 

Muit/? " .29* CTBS Posttest - l.l'X^ + .SOXj 

•p<.01 
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Analygia and resiilts . Table 6 shows group means and standard 
deviations of the pretest and posttest measures. The NRS group 
started out slightly ahead of the basal group in first grade and ended 
up only slightly ahead at the end of second grade. The regression 
analysis used a dunnmy variable for NRS (!)• Table 6 presents the 
correlation coefficients, the multiple R, and the raw B weights for the 
regression solution. Again the B weight for NRS is not significant but 
significance was obtained for pretest, F (1, 109) = 9, 16, as well as 
the overall Mult R. 

Process data for the grade two. District B classrooms were 
collected during 1977-1978 via a questionnaire completed by the 
teachers. These data are identical for Studies 5 and 5a. The oppor- 
tunity and structure information is similar to previous years. In 
terms of opportunity , NRS had fewer students, more adixlts, equivalent 
attendance, and lower allocation of reading time. Variables within the 
structure construct reveal that in NRS classrooms* students were .aore 
often self-paced, used more games, had faster feedback, had individ- 
ualized assignments, were tested more frequently, and did better on 
teacher tests. 

By second grade, some variables in the content construct have 
taken on a different look as would be expected. For example, second 
grade NRS students engaged in letter-sound correspondence, letter 
identification, and sotmd blending less often than the basal students 
and less often than they did in first grade. This can probally be 
explained by the fact that, in NRS, most of those kinds of exercises 
are taught in the first few levels, usually covered in the first grade. 
Second grade NRS teachers rate the curriculum as closely matching 
what they consider to be important second grade reading objectives 
(4. 5), while non-NRS teachers rate their curriculum less well (3. 5). 
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Across^Digtrict Contrast; Study 6 

The final study to be presented here involves a contrast between 
two school districtst District A, which was using NRS in first grade, 
and District Cp using Scott-Fcresman or Economy, both of which are 
basal reading curricula. The study took place during 1976-1977 and 
represents the third ring in the set of contrasts. District A was 
implementing NRS in all its first grade classrooms that year; as 
described earlier, a second district was identified for use as a com- 
parable contrast group. This contrast provides a more rigorous 
control for the effect of the innovation. Within District A, NRS was 
clearly identified as the innovation, and it was possible that the non- 
NRS teachers were distressed at not having it. District C teachers, 
in contrast, did not know of the existence of NRS. 

Population * Census tract data from 1970 revealed similarities 
between District C and District A in terms of SES and race. By early 
1976, when plans were being drawn up to establish District C as a 
control, the population in District A had begxui to change. Three 
classrooms in two schools in District C closely resembled those in 
District A vnth regard to SES and ^)retest information, so they were 
used to represent the contrast group for the eight District A classrooms. 

Method. Both groups were tested in the fall, 1976 vnth the Murphy- 
Durrell Reading Readiness Analysis, and in the spring, 1977 with the 
Stanford Achievement Test. The Letter Names subtest from the Murphy- 
Durrell was used as the pretest measure and the Total Read score from 
the SAT was the criterion measure. 

Analysis and results . Table 7 shows means and standard devi- 
ations for the groups on both pre- and postteats* NRS students start 
out slightly below the basal students, but end up well ahead of them. 
The NRS group scored 2 points below the non-NRS group on the pre- 
test, but 12.3 i>oints above the control grcnp on the posttest. End- 
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of-year achievement for both groups was regressed on pretest and a 
dummy variable code for NRS (1). Table 7 shows the correlation 
matrix of the variables in the regression, the multiple R, and the raw 
B weights for the regression solution. Both B weights (pretest and 
NRS, 2.1 and 16. 5) are significant, F (1, 238) = 77.42 and 25. 04 
respectively. 

Data regarding classroom practices were collected with an inter- 
view of the eight NRS District A teachers and the three comparison 
District C non-NRS teachers. These data are presented in the last 
column of Table 2. Variables in the opportvinity construct show that 
District A classrooms had an equivalent number of students enrolled, 
adults in reading, and percent of students present. District A had 
fewer minutes allocated to reading (200 minutes less per week), and 
slightly lower percentage of overlap. 

Table 7 

^ Study 6: 1976-1977 Acrosi Dlstric t Contrast 

Achievement Means and Standard Deviations of NRS and non-NRS Groups: Matched Cases 



Pretest: Murphy-Durrell 


NRS (170) 


Non-NRS(71) 




Letter Names Total 


44.57 


46.61 


X 






6.12 


S.D. 


Posttesti SAT 








Total Read 


100.89 


88.61 


>r 




26.60 


26.28 


S.D. 


Correlation and Regression {N 


-241) 




1) Murphy-Durrell pretest 


1 
X 


2 


3 


2) NRSd) 


-.14 


X 




3) SAT Posttest 


.45* 


.21* 


X 


Mult/; -.53* 


SAT-ri 


•X, + 16.5*X2 





•p<.oi 
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In the structure conitruct, NRS teachers unanimously reported 
that their classrooms were self-paced and that they used more games. 
NRS students got work corrected and returned to them more quickly, 
went for far fewer days between tests, and performed slightly better 
on the tests* though the difference between groups was not dramatic. 
Most of the students in the District C group were assigned the same 
lessons as their classmates, while 90 percent of the students in 
District A were assigned unique lessons. 

The content construct reveals that the NRS group spent the same 
amo\int of time engaging in letter-sound correspondence, slightly less 
time identifying letters, and the same amount of time blending. District 
A teachers rated the match between NRS and what they thought to be 
important first grade reading objectives as 4.5, while District C 
teachers rated the match as 2.5. 

Discussion and Implications 

The effects of NRS were assessed through several related eval- 
uations. £ach new study contributed more Information. The contrasts 
can be considered concentric rings expanding out to more distant 
settings from the central starting point. Initial settings were geo- 
graphically close » involved considerable support during the imple- 
mentation process, and used criterion measures that had a tight 
relationship to what was taught. Later contrasts used settings that 
were further away, involved less support, and used more general 
criterion measures. 

After the initial field test of NRS in LRDC-affiliated developmental 
schools* negotiations to implement the new curriculum in a school 
unrelated to LRDC were completed, and the first within-school con- 
trast was established. Concerns of developers and school adminis- 
trators led the program into further implementation within that school 
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diltrictt and the aeries of wlthin-diBtrict contraata waa begun. The 
aucceaa of NRS In thoae two aettlnga led to the move complex contraat 
acroaa dlatrlcta. The poaitive ireaulta of moat of the etudlea conducted 
\inder very different* Increaalngly demanding clrcumatancea^ with less 
and lea a implementor aaalatance^ apeak well of the Iterative » concentric 
approach to implementation and evaluation. 

The data aupport NRS aa a aucceaaful flrat grade reading program. 
Four atudlea ahow NRS atudents performing conalderably better than 
their counterparta ualng a variety of baaal aeries (Scott-Foreaman, 
Harper and Row» Houghton- Mifflin » and Economy). One study of 
second grade atudents in one district did not show NRS students to have 
gained more than non-NRS students. We analyzed the second grade 
data In two ways: flrat» a simple pre/post contrast; second^ a longi- 
tudinal study of thoae students that were in either program for two 
years. This aecond contrast* while obviously suffering from selection 
bias, waa important In case we were observing students that had had 
only one year of NRS. One poaaiblUty Is that many of the students in 
NRS in the aecond grade had moved out of NRS and were In a different 
curriculum. However* this Is only speculation. Further research Is 
clearly needed to evaluate the effectiveness of NRS at the second grade. 
Five seta of proceaa data were collected over the years. General 
trends noted from the data are summarized. 

Opportunity . NRS classrooms tend to have fewer students enrolled 
^knd more adults present during reading instruction. However* in 
Study 2* the number of adults la almost equal. The percentage of 
students present tends to be equivalent for NRS and comparison grouptr. 
The number of minutes per week allocated for reading instruction 
varies. It appears that even when NRS students have less time avail- 
able, they usually perform better. VThen the overlap estimate between 
what was taught and what was tested on the end-of*year achievement 
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meaiure wfti collected* the NRS group had a higher percentage of 
overlap in two out of three cases. 

Structure , It Is evident that NRS students were aelf-paced and 
used games as a part of reading Instruction far more frequently than 
their non-NRS counterparts. NRS teachers corrected and returned 
work to students more quickly and made assignments on a more 
Individualised basis. NRS students were tested more frequently and 
performed better on those tests than did the basal students^ Indicating 
that thoy were tested on material they had learned. 

Content . First grade NRS students spent more time than basal 
students and second grade NRS students engaging in letter- sound 
correspondence, letter Identification, and blending. This trend Is 
most probably a function of the fact that the NRS curriculum presents 
those kinds of activities early in the first grade. NRS teachers, 
across all of the studies, rated the match between the curriculum and 
what they considered to be important first or second grade reading 
objectives as a close match. 

This paper used a set of sequential studies on the effectiveness 
of NRS as an example of a strategy for curriculum evaluation. These 
studies follow the natural development and expansion of a curriculum 
program and provide convergent evidence for the value of a particular 
innovation, NRS. None of the studies alone could provide convincing 
evidence for the value of the curriculum, and even this collection of 
studies still leaves questions unanswered. We do know that the program 
can be implemented successfully in several very different settings 
with decreasing amounts of developer support. We can also see that 
NRS is very successful in fi rst gr ide classrooms on several different 
dependent measures (two versions of the SAT and the CTBS). However, 
the one study of NRS at the second grade did not provide evidence for 
its success there. The strategy of cumulating several small studies 

29 



33 



In multiple settings is clearly feasible and follows the natural sequence 
of development and implementation* 

The strategy of iterative evaluation is not offered as a replacenient 
for the two more prominent approaches currently available^ experiments 
and explanatory observational studies* Rather^ iterative evaluation is 
seen as an alternative to waiting for the perfect evaluation conditions to 
prevail or to ignoring the natural production of evaluative information 
that accompanies program development* Whether it is a desirable 
alternative remains to be seen* 
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